Curiously Fast Convergence of some Stochastic Gradient Descent Algorithms

1 Context Given a finite set of m examples z 1 ,. .. , z m and a strictly convex differen-tiable loss function ℓ(z, θ) defined on a parameter vector θ ∈ R d , we are interested in minimizing the cost function min θ C(θ) = 1 m m i=1 ℓ(z i , θ). One way to perform such a minimization is to use a stochastic gradient algorithm. Starting from some initial value θ[1], iteration t consists in picking an example z[t] and applying the stochastic gradient update θ[t + 1] = θ[t] − η t ∂ℓ ∂θ ℓ(z[t], θ[t]) , where the sequence of positive scalars η t satisfies the well known Robbins-Monro conditions t η t = ∞ and t η 2 t < ∞. We consider three ways to pick the example z[t] at each iteration: • Random Examples are drawn uniformly from the training set at each iteration. • Cycle Examples are picked sequentially from the randomly shuffled training set, that is, z[km + t] = z σ(t) , where σ is a random permuta-• Shuffle Examples are still picked sequentially but the training set is shuffled before each pass, that is, z[km + t] = z σ k (t) , where the σ k are random permutations of {1, .